Inside-Outside Estimation Meets Dynamic EM

We briefly review the inside-outside and EM algorithm for probabilistic context-free grammars. As a result, we formally prove that inside-outside estimation is a dynamic-programming variant of EM. This is interesting in its own right, but even more when considered in a theoretical context since the wellknown convergence behavior of inside-outside estimation has been confirmed by many experiments but apparently has never been formally proved. However, being a version of EM, inside-outside estimation also inherits the good convergence behavior of EM. Therefore, the as yet imperfect line of argumentation can be transformed into a coherent proof. 1 Inside-Outside Estimation The modern inside-outside algorithm was introduced by [4] who reviewed an algorithm proposed by [1] and extended it to an iterative training method for probabilistic context-free grammars enabling the use of unrestricted free text. In the following, y1 . . . yN are numbered (but unannotated) sentences. Definition: Inside-outside re-estimation formulas for probabilistic context-free grammars in Chomsky normal form are given by (see [4], but see also [1] for the special case N = 1): p̂(A → a) := ∑yN w=y1 Cw(A → a) ∑yN w=y1 Cw(A) , and p̂(A → BC) := ∑yN w=y1 Cw(A → BC) ∑yN w=y1 Cw(A) . The key variables of this definition are so-called category and rule counts: Cw(A) := 1 P ∑n s=1 ∑n t=s e(s, t, A) · f(s, t, A), Cw(A → a) := 1 P ∑ 1≤t≤n, wt=a e(t, t, A) · f(t, t, A), and Cw(A → BC) := 1 P ∑n−1 s=1 ∑n t=s+1 ∑t−1 r=s p(A → BC)e(s, r, B)e(r + 1, t, C)f(s, t, A) which are computed for each sentence w := w1 . . . wn with so-called inside and outside probabilities: An inside probability is defined as the probability of category A generating observations ws . . . wt, i.e. e(s, t, A) := p(A ⇒ ∗ ws . . . wt). In determining a recursive procedure for calculating e, two cases must be considered: • (s = t): Only one observation is emitted and therefore a rule of the form A → ws applies: e(s, s, A) = p(A → ws), if (A → ws) ∈ G (and 0, otherwise). • (s < t): In this case we know that rules of the form A → BC must apply since more than one observation is involved. Thus, e(s, t, A) can be expressed as follows: e(s, t, A) =